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The writing test at Colorado State University 
determines which students move from basic composition into the 
college composition course. Initially, all incoming students take a 
placement test similar to the holistically scored writing tests 
administered by the Educational Testing Service. This placement test, 
along with other information, provides generalizations about the 
students 1 skills and deficiencies. Once appropriately placed, 
students in the basic writing course find themselves focusing on 
writing skills — generating ideas, organizing them to communicate 
clearly to readers, developing them fully to convince readers, and 
editing them to reduce distractions. As a pretest at the beginning of 
the semester, students write an essay on an accessible topic in one 
hour. At the end of the semester, students are asked to write on a 
parallel topic. Because of the pretesting and posttesting, 
instructors can compare skills at the beginning and end of the 
semester and can evaluate skills that are expected of any student 
completing the basic course. Failing tests are examined along with 
other writing the weak students have produced during the semester. If 
students fail the final exam because of limited but consistent 
problems that they have begun to master in the final weeks of the 
semester, they are usually passed with hopes that they will master 
specific skills early in the next semester. Instructors of subsequent 
composition classes have applauded this testing process since it 
guarantees them more homogeneous classes. (HOD) 



********************************************** 

* Reproductions supplied by EDRS are the beit that can be made * 

* from the original document. * 
******************************************* **************************** 



vO 

0 s Testing Basic Writers' Proficiency: 

An Effective Model 

K\ 

c\] 

Q 

Kathleen E. Kiefer 



U.S. DEPARTMENT OF EDUCATION 
NATIONAL INSTITUTE OF EDUCATION 
EDUCATIONAL RESOURCES INFORMATION 

CENTER IERICI 
^This document has been reproduced as 
received from the person or organization 
originating it. 
! 1 Minor changes have been made to improve 
reproduction quality. 

• Points of view or opinions stated in this docu- 
ment do not necessarily represent olficial NIE 
position or policy. 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 

Kathleen E. Kiefer 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



ERLC 




Testing Basic Writers" Proficiency's 
An Effective Model . 

Teachers of basic writing are often called upon to demonstrate 
the effectiveness of basic writing courses and to guarantee 
students* ability to succeed in subsequent writing courses- 
While we may not happily accept the burden of competency 
testing.,, we often find ourselves forced to establish such 
tests and thus must concern ourselves with devising fair, 
reasonable, and effective competency tests,, First, as writing 
teachers, we must be concerned with finding a test of WRITING 
skills. Although results from standardised tests of grammar 
and usage correlate well with ultimate success in academe,, 
writing teachers should not accept such tests as adequate 
measures of writing skills. One critic of standardized testing 
points out that the number of televisions in a student's home 
or the family income correlates equally well with academic 
success- Moreover , most writing teachers have unearthed 
bright students who score well on standardized tests but who 
can't write coherent paragraphs* If our testing reflects our 
emphasi s on produci ng focused , f ul 1 y detai 1 ed , and coherent 
texts, then we must test writing and not recognition of error 
or accepted usage* 

Another difficulty for testers is to articulate clearly just 
what skills are being tested, to define criteria appropriate 
for students : ' needs and for the curri culum. Furthermore, 
although we spend little time determining validity and 
reliability of tests we give in composition, these factors 
^ should be important to us. A good competency test should 
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discriminate well between those who have mastered clearly 
defined writing skills and those who have not. A valid test 
will give similar results time after time- If possible, a 
sound competency test should represent a variety of writing 
tasks common in college or workplace writing so that students 
are assured of the practical value of their writing 
instruction. Students will react more positively to 
competency testing if they see that the skills required are 
skills the work world acknowledges as important« 



And beyond these general concerns for valid or appropriate 
testing, teachers must also consider what happens to students 
who fail the test- If failing a writing test prohibits a 
student from graduating, and if the student has no recourse 
but to drop out of school;, then the writing test will surely 
be scored on a different scale than the test that determines 
whether students pass on to a second composition course in a 
thr ee~semest er sequence- 

In addition to these general concerns, at Colorado State 
University we are also faced with practical, administrative 
concerns- First,, our graduate teaching assistants are drawn 
from Master's programs and thus teach for us for only two 
years. We supervise them closely and need a competency test 
that relieves them of the burden of failing students who 
clearly do not meet our standards. Inexperienced teachers 
often falter when asked to fail a student who has worked hard 
made progress, but sti 1 1 cannot meet standards. Moreover, 
because CSU has only one composition course required of all 



undergraduates at the university., we need to assure all 
•faculty that students entering the college composition course 
from the basic writing course will not hinder their 
cl assmates- 

Gur test is not a university requirement -for graduations 
rather, it determines only which students move -from basic 
composition into our college composition course- As I will 
make clear, though, the testing method effectively answers the 
concerns I have outlined while providing additional 
pedagogical benefits- 

Before describing our testing method., let me provide some 
necessary background on the composition program at CSU, 
including our initial placement exam:, the profile of students 
placed into basic writing (C0101),. and the focus of the 
course. All incoming students--*- reshmen and transfer students 
lacking credit in composition — take a placement test similar 
to the hoi isti call y scored writing tests administered by ETS, 
At CSU, students read a selected passage on a topic of general 
knowledge, summarise- the main points in the passage., and then 
develop a response agreeing or disagreeing with the main 
points. The scoring criteria for this placement test clearly 
discriminate students at the top and bottom of the scales 
students who summarise completely and clearly;, who state a 
clear thesis, and who develop a thorough response typically 
test out of the composition sequence at CSUii students who only 
summar i ze, who cannot organi ze or devel op thei r responses, or 
who have repeated, serious errors are placed into our basic 
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composition class. About 67. of the students taking the test 
in any year test out of composition altogether; some 23% are 
pi aced i nto the fundamental s course each year „ 



This placement test, combined with other information, allows 
us to generalize about our students" skills and deficiencies. 
When we consider relatively objective measures of students-" 
preparation for college composition,, we note that students 
placed into our basic writing course have SAT verbal scores 
ranging from 200 to 640, high school English averages from 1.7 
to 4.0, and percentile rankings from 30 to 100 (the latter 
being the valedictorian of the class). Students placed into 
our college composition course have similar scores, though 
some students have higher SAT verbal scores. Such scores 
alone simply do not distinguish students who wi 1 I benefit from 
a basic writing class from those who need very little. work on 
academic writing to succeed at the university. Yet when we 
ask students about their background as writers, we quickly 
develop a profile of the unpracticed writer for students 
placed into C0X01. On questionnaires, these students commonly 
report having taken media, broadcasting, and theatre courses 
(stagecraft) in lieu of high school English courses with 
writing as a major component. Perhaps even more disturbing., a 
significant number of these students report having read only 
one or two books during their high school years — although most 
read one or two books a year during high school — and most tell 
^ us that SPORTS ILLUSTRATED and SEVENTEEN constitute their 

regular reading. About half have written analytic prose in 
O their senior year in high school; and of those, the most 

ML 



common assignment has been a "research" paper graded more on 
form than on content. Although our basic writers are by no 
means as deficient as the typical writer Mina Shaughnessy 
describes in ERRORS AND EXPECTATIONS, neither are they ready 
•for college composition. 

Once appropriately placed? students . in the basic writing 
course find themselves focusing on writing skills — generating 
ideas? organising them to communicate clearly to readers? 
developing them fully to convince readers, and editing to 
reduce distractions. Students write paragraphs of 100 to 300 
words every week, practicing strategies for developing their 
ideas. Because we have focused on specific writing skills we 
expect our basic writing course to teach, we are able to 
design a test that measures students' 1 mastery of these skills 
and then evaluate careful 1 y weaker students*" progress at 
semester * s end - 

Let me describe our testing model briefly and then discuss its 
valuable features. Students write a pre-test on an accessible 
topic in one hour at the beginning of the semester* 
Generally? we have two parallel topics for the pre-test, best 
and worst teachers, for instance« One half of the students 
write on one topic?" the other half write on the opposite 
topic. At the end of the semester, students write on the 
parallel topic they did not see at the beginning of the 
semester. We code pre-tests so that scorers cannot 
distinguish pre- and post-tests, mix pre- and post-tests, and 
score all tests in a sirgle three to four hour session. We 
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use a six-point scale with clearly de-fined criteria for each 
point and range in the scale, and we distribute model 
paragraphs for each range- In all holistic scoring sessions, 
we standardize carefully before and during the scoring. 

Why do we feel that our competency test is effective? We see 
clear benefits of our test design, consistent scoring, and 
results that satisfy our concerns outlined at the beginning 
of this paper - 

Through pre-testing, we can examine new topics as valid 
measures of the skills taught in the course. One of our 
measures for the accessibility of topics involves reading a 
percentage of the pre-tests to determine typical problems 
students had with the topic A small group of experienced 
teachers and graders reads perhaps ten percent of the 
diagnostic paragraphs drawn randomly from all pre-tests* If 
students seem to flounder because of the topic or wording of 
the assignment, we revise the topic before administering it 
again* Only by screening topics can we feel confident that we 
have removed unduly difficult ones. 

As we read pre-tests, we also consider how we might revise the 
criteria for cur ho 1 , istic scoring exercise. We choose model 
paragraphs and samples that illustrate the range of students 3 
skills. We also anticipate the training process — 
standardising — and select samples that will illustrate 
appropriate and inappropriate responses to the topic 
I+ " we know from past experience that readers react positively 



or negatively to a particular response to an assignment, as 
■for instance the occasional inability to -follow directions 
precisely, we choose at least one sample paragraph so that we 
can discuss an appropriate response with the entire group of 
scorers. 

As a final benefit of our test design.,, the pre-tests help us 
guard against a weakness inherent in "the holistic method of 
scoring papers. Readers often balk at using the entire range 
of scores available to them- In particular? scorers hesitate 
to Assign the highest and lowest scores- When we have not 
mixed pre- and post-tests, we have noticed a strong tendency 
to assign only mi dd.l e-range scores- By mixing pre- and 
post-tests, we know that at least one-third of the papers 
should fall into the lower range of scores- Scorers presented 
with many poor papers soon overcome thei r reluctance to assign 
lower-range scores- In contrast, well-written papers stand 
out so that they easi 1 y receive high scores :i thus assuring 
that the entire scoring scale gets used « 

In addition to the benefits of this test design, our 
competency test allows us to gauge students*" progress, not 
subjectively through analytic scoring, but objectively- Our 
model provides two standards of comparison: because of pre- 
and post-testing, we can compare skills at the beginning and 
end of the semester and because we collect data from one 
semester to the next, we can evaluate ski lis we expect of any 
student completing the basic course- Specifically, when we, 
compare pre- and post-test means, students average between 1.8 
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and 2.1 on the pre-test and between 3« 5 and 3,7 on the 
post-test. Students, thus, typically gain between l.S and 2 
■full points over the semester 9 a significant gain on s 
six-point scale. We have noted these gains over the full 
three years of using the testing procedure. Furthermore;, we 
have computed the reliability of these scores by having 
readers rescore samples used from one grading session to the 
next and by computing the inter-rater reliability of scores 
assigned to the same paper. Our GTAs find the pre-tests 
slightly more difficult to score than the post-tests! they 
consistently assign more pre-tests split scores so that we 
must read between 17 and 22 percent of pre-tests a third time 
(for IRR of 78 to 827.). On post-tests, two scorers agree on a 
score between 85 and 92 percent of the time. 

A second measure of successful scoring appears in the rate of 
students failing the test and failing the course. Since we 
began administering a holistical ly-scored "final exam 11 or 
competency test, 18 to 207. fail the test and 127. fail the 
course. Only in spring semester 1983 did we see any 
significant change in this measures only 147 failed the test 
and 87 failed the course. We have used the topic successfully 
in the past and so cannot attribute this drop in failure rate 
to an exceptionally easy topic. We did introduce 
computer-assisted editing for students and so must test 
further to see if that aid accounts for greater student 
success in the course. 



A final major success for our testinn nrnnram mm&cz -f^r-.m fKn 



limitations we sat f or it- We do not expect it simply to cull 
poor writers so that they do not pass the course. Rather, we 
pull out failing tests and then consider other writing weak 
students have produced during the semester- Ever- failing 
final paper is reviewed by the instructor and the supervisor 
of the program. Students who have had an "off" day are easy to 
spot, mainly because we can look at other impromptu writing 
samples. But we also consider students' edited work, 
attitudes, and achievement. If students fail the final exam 
because of limited but consistent problems that they have 
begun to master in the final weeks of the semester, we will 
usually pass those students in hopes that they will master 
specific skills early in the next semester. Each case, 
however, is unique, and we must consider both the strengths 
and weaknesses of each student. The GTAs feel comfortable with 
this process because they often find themselves unduly 
influenced by emotional reactions to the student — just as 
often negative as positive. By reviewing students 3 work with 
me, instructors often see how to help basic writers achieve 
success in our course or other basic writing courses they will 
teach. 

I must mention other positive reactions to our testing model. 
Instructors of the subsequent composition class applaud our 
testing process because it guarantees them more homogeneous 
classes. Perhaps even more important, students themselves 
react positively. They support the notion that all students 
taking the course are judged according to the same standards, 
a policy not found in mnqt- nf-h 
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CSU or ether institutions. They appreciate the 
"accountability 11 the test imposes on their instructors and on 
them,, and they praise the objectivity of scoring because they 
see that more than one instructor contributes to the final 
score, Finally, while students -frequently disparage the basic 
writing class while they are taking it, they praise the course 
oner? they move to the subsequent courses, C0150, in part 
because the test gives them confidence as writers. Moreover, 
unlike students who place out of C0101, students who have 
passed the competency test after taking C0101 have mastered 
skills that help them achieve higher grades in C0150 than 
students i^ho do not take COIOI. 

Our test, then, is reliable and effective for students, 
instructors, and administrators. With each an objective 
measure of students 7 and teachers 1 * success, we have been able 
to convince any who question our writing program that we have 
set attainable but demanding writing goals we can help 
students meet. 



